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Abstract 

Since “the year of the MOOC” in 2012, the effectiveness of massive open online courses (MOOCs) has 
been widely debated. Some argue that MOOCs are not an effective mode of instructional delivery because 
of low completion rates. In the interest of developing alternative indicators of performance this study 
draws from recent efforts to measure engagement in social media, as well as from research on indicators 
of student engagement in traditional college courses. Using data from 16 Coursera MOOCs offered by the 
University of Pennsylvania we calculate standardized access rates for lectures and assessments. While 
these indicators have clear limitations as measures of educational progress they offer a different, more 
nuanced understanding of the level and nature of users’ engagement with a MOOC. This paper shows that 
a very small share of users takes up available opportunities to access course content but notes that the 
standardized access rates compare favorably with those for social media sites and with response rates to 
large-scale direct mail marketing programs. For MOOC providers and platform managers, indicators like 
the ones developed in this study may be a useful first step in monitoring the extent to which different 
types and combinations of activities may be providing better opportunities for learning. 


Introduction 

Since “the year of the MOOC” in 2012, the effectiveness of massive open online courses 
(MOOCs) has been widely debated. Some argue that MOOCs are not an effective mode of instructional 
delivery because of low completion rates. For example Pema et al. (2014) found fewer than 10% of those 
registered for “first-generation” MOOCs at the University of Pennsylvania completed the course. 
Completion rates were low, regardless of whether users followed the instructor’s suggested pattern of 
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experiences or chose the order to access learning materials. Other studies have reported similarly low 
completion rates (e.g., MOOCs @Edinburgh Group, 2013; Ho et al, 2014). 

Using completion rates to gauge performance is consistent with conventional approaches to 
quantifying student behavior in a course of study. Enrollment, progress, and completion were commonly 
measured characteristics of early models of education systems (Stone, 1970; King, 1972) and enrollment 
and progress data recur in the OECD’s Education at a Glance (OECD, 2014) statistical series. 

Completion of a course is only a partial indicator of a MOOCs’ performance or student 
outcomes. Some advocates, such as Koller et al. (2013), stress low completion rates can still result in 
large numbers of beneficiaries, given the very high number of users who register for a course. Moreover, 
completion may not be a MOOC user’s goal (Koller et al., 2013) and focusing on completion could 
obscure the possibility that intermittent interaction with a MOOC may produce user benefits (Haggard, 
2013; Ho et al., 2014). Conventional indicators of progress may be “ill-suited” to the environment or 
course design of a MOOC (Hoffman and Fodor, 2010, p. 42), as MOOCs allow episodic and partial 
involvement as well as ordered and sequential movement within a course. 

Recognizing that measures of completion are “fairly blunt instruments when it comes to making 
sense of student interaction, engagement and learning,” Coffrin et al. (2014, p. 85) focused on identifying 
categories of users in MOOCs, based on users’ levels of engagement in course content. Their analyses 
revealed three categories of users: “auditors”—those who watch content but do not do assessments; 
“active” learners—those who completed at least one assessment; and “qualified” learners—those who 
watched content, completed an assessment, and scored well. This effort to categorize users in MOOCs 
aligns with attempts to sort social media users along a continuum of “visitors,” i.e., those who see the 
Internet as a tool, and “residents,” those who use the internet as a social space (Wright et al., 2013; White 
and Le Cornu, 2010). 

This study explores patterns of engagement in MOOCs, drawing from efforts to measure 
engagement in social media, as well as from research on indicators of student engagement in MOOCs and 
traditional college courses. In particular, our interest is in creating a measure of the rate of taking up 
MOOC learning opportunities that is easy to calculate and interpret. We explore the utility of several 
potential indicators of engagement in a MOOC by using data from 16 Coursera MOOCs offered by the 
University of Pennsylvania. 

Indicators of engagement akin to those used in social media may be useful because of similarities 
between MOOCs and social media sites. For instance, MOOCS and social media use “web based 
technologies to create...interactive platforms via which individuals and communities share...(c)o-create, 
discuss and (possibly) modify...content” (Keitzmann et al., 2011, p. 241). The technology on which 
MOOCs and social media sites depend makes communication among users easy and inexpensive. Both 
MOOCs and social media sites are usually freely accessible and allow users to access the sites when they 
wish; the user controls the duration, rate, frequency and order of interactions. Both offer opportunities for 
different levels of activity allowing users to exchange information and view, manipulate and create 
content. Both put a premium on reaching large numbers of people and retaining their interest. 

Social media sites can be thought of as learning environments or “passionate affinity spaces” 
(Gee and Hayes, 2011, p.69) where people with shared interests exchange ideas and information freely, 
regardless of prior experience and expertise, age, gender, location or wealth (Gee 2013a, p. 175). Social 
media sites use a “participation metaphor” (Sfard 1998, p. 6) of learning and engagement where the goal 
is community building and the emphasis is on belonging to the group and participating in its activities. 
While the Coursera MOOCs share some of the characteristics of affinity spaces, many of the courses we 
study emphasized individuals as recipients of content provided by experts with a goal of mastery, more 
akin to Sfard’s metaphor of learning as acquisition (p. 7). Some courses like Modem Poetry (MoPo) 
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placed great weight on participation, peer review, collegiality and exchange. As Sfard argues there is a 
“case for the plurality of metaphors” (p. 11) and, when the technology is still evolving, as in the case of 
MOOCs and measurement of MOOC effectiveness, exploring activity measures like those used by social 
media is a potentially fertile field of inquiry. 

Engagement 

The literature on engagement and learning illustrates the complexity of these constructs. 
Azevedo (2015) points to the scope and diversity of approaches to delineating and quantifying 
engagement. In his commentary on six articles representing “the various theoretical, methodological, and 
analytical challenges,” Azevedo begins by declaring that “engagement is one of the most widely misused 
and over-generalized constructs” (p. 84). Azevedo concludes by arguing for greater attention to “process 
data” as this “will lead to advances in models.. .analytical techniques and.. .instructional 
recommendations” (p. 93). 

Reflecting an interest in process, higher education institutions commonly emphasize 
“engagement” as a mechanism for improving the college experience and increasing graduation rates (see 
Price & Tovar, 2014). Attention to engagement is grounded in the work of scholars such as Tinto (1975, 
1987), Astin (1984) and Kuh and his colleagues (2006) who identify behaviors that arguably enhance 
learning. Engagement, for example, can be defined as the “time and effort students devote to activities 
that are empirically linked to desired outcomes.. .and what institutions do to induce” participation in these 
activities (Kuh, 2009b, p. 683). Engagement is assumed to be a key mechanism for promoting learning as 
it “builds the foundation of skills and dispositions that people need to live a productive satisfying life” 
(Kuh, 2009a, p. 5). It includes interactions with learning materials, other learners, faculty, and co- 
curricular activities. 

Student engagement research often focuses on on-campus and face-to-face instructional activities 
but some scholars have considered the relationship between engagement and outcomes in on-line courses 
(Coates, 2007; Chen, Gonyea, & Kuh, 2008; Robinson & Hullinger, 2008), in learning management 
systems (Beer et al., 2010) and in social networks (Thoms and Eryilmaz, 2014). As an example, Robinson 
and Hullinger (2008) used a version of the National Survey of Student Engagement (NSSE) to explore 
variations in engagement of students enrolled in at least one completely on-line course. The results 
suggest variations in engagement are associated with students’ academic performance, academic major, 
and age, and faculty creating “purposeful course designs that promote interaction, participation, and 
communication in the online learning environment” (Robinson & Hullinger, 2008, p.107). 

Although measured much more simply than in on-line or e-leaming environments, engagement is 
also a key criterion for assessing the effectiveness of social media sites as marketing tools (Social Bakers, 
2013). Each visit to a site or a page is an engagement, for instance, and represents an opportunity to place 
new content before a potential purchaser. This type of engagement measure is used to assess the 
effectiveness of marketing strategies for products as diverse as luxury shoes (PR newswire, 2013) and 
furniture (French, 2012). 

Although social media networks offer different opportunities for engagement (Dunham, 2014), 
many use a common approach to assess engagement. The goal is capturing consumer actions and 
behaviors beyond the simple act of viewing content (Hollebeek, Glynn & Brodie, 2014). These 
approaches assume that success on social media platforms like Facebook is “not primarily a matter of 
amount of fans.” Activity is more important: “likes, comments and shares are the fuel of vitality” (Eyl, 
2013, p. 1). User engagement with content on a social media site is “likely to generate commitment,” 
brand loyalty, and repeat business (Hoffman & Fodor 2010, p. 46.) 
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Typically, approaches to measuring engagement in social media aggregate the number of “Likes,” 
“Posts,” and “Shares,” or things like replies on Twitter and comments on Linkedln and Instagram, for a 
specific period. This number is then related to the number of followers or fans during the same period. 
Averaging these rates across days or weeks produces a “mean engagement rate.” Calculating activity per 
user, for instance, recognizes the dynamic nature of social media and is an indicator of how many people 
are “connecting with your brand and how often” and how many times they draw attention of others to the 
content or item (Ken, 2014, p. 2). 

Engagements rates make it easier to understand the large and complex activity of many social 
media sites, partly by sifting out one-time visitors and passive followers. In an analysis of more than 4.9 
million posts over three months on 60,000 Facebook pages, Eyl (2013) found most sites have “very low” 
engagement rates. Half of all sites had rates of “0.002 interactions per fan per day.” In essence, 
engagement rates in social media are the sum of individuals’ discrete actions that change a screen view, 
standardized to some measure of the population. This standardization allows comparisons of engagement 
rates between courses or segments of courses. 

While there is a commercial driver behind much of the attention to engagement rates in social 
media settings, the activities measured are essentially acts of participation, with the user doing something 
in relation to content on a site. This approach echoes Sfard’s (1998) idea of seeing learning in terms of 
“participation,” as individual “learners contribute to the existence and functioning of a community of 
practitioners” (p. 6). It also indicates how large numbers of individuals organize or “self-regulate” 
learning activities which “is of particular importance in relatively open learning environments.. .where 
learning is enhanced by digital technologies” (Steffens, 2015, p. 49). 

Applying Engagement Rates to MOOCS 

Measures of engagement in social media are less sophisticated than measures that have been 
developed to understand engagement in on-campus courses and some on-line courses. Nonetheless, 
applying the basic notions of social media engagement metrics to MOOCs may provide useful insights 
into the activities of MOOC users. The first generation of Coursera MOOCs was usually neither credit¬ 
bearing nor credentialed. They were less formal learning environments with flexibility in terms of time 
spent or required, order and rate of progress through materials, and involvement in assessment activities. 
As such, they are akin to the largely informal online learning environments like social media sites and 
digital games and fan sites which “connect playing and learning to social interaction and mentoring” 
(Gee, 2013b, p.18). Although MOOCs use formal, frequently didactic, instructional methods, they operate 
sufficiently like fully informal environments like Facebook or Instagram to make the use of social media 
metrics worth exploring. 

This study answers the question: how often do users access content, be it a course lecture, a quiz, or 
another assessment? Applying measures from social media to address this question accommodates 
asynchronous access and differences in an individual’s intentions, ft also places user behavior at the heart 
of understanding the extent to which users access large-scale open online course content. This study does 
not classify users into “types,” assuming that users may behave differently at different times and 
circumstances. In this study, the access rate quantifies the act of opening a lecture or quiz, activities that 
precede and may presage learning. We call these measures ‘access rates’ to avoid inferring any user 
behavior beyond the act of opening a page view. We are measuring the frequency of a basic activity, how 
often the individual’s screen view changes, that is a precursor to constructive and interactive learning 
activities like generating something that is not in the material presented or confirming or debating 
another’s proposition (Chi 2009, pp.77-83) 
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An Example: Access Rates for Penn’s First-Generation Coursera Offerings 

We used this approach to examine user access in 16 Coursera courses taught for the first time by 
faculty at the University of Pennsylvania between June 2012 and July 2013. The courses varied in field of 
study, design, length and number of access opportunities. 

We consider the extent to which users access four types of instructional materials: lectures; 
quizzes embedded in lectures; stand-alone quizzes; and open-ended questions. Our approach gives equal 
weight to different activities—that is, accessing a lecture counts the same as responding to an open-ended 
question or attempting a multiple-choice quiz. This approach avoids making assumptions about the type 
of activity that “best” promotes learning in an online environment although Chi’s (2009) taxonomy of 
active, constructive and interactive learning suggests simply viewing a lecture is a less engaged activity as 
compared to identifying the right answer or creating original text. We do not consider activities such as 
posting a message to a discussion forum, as Coursera did not provide us with access to these data and 
because of challenges associated with interpreting these data (as discussed by Brinton et al., 2013). We 
focus instead on exploring indicators of engagement that instructors can easily adopt and that can be 
replicated across large numbers of courses at little cost. We do not perform statistical tests as the analyses 
describe data from the population, not a sample. Moreover, with the high number of cases in the analyses, 
all statistical tests would be statistically significant. 

Two Indicators of User Activity 

In what follows we use two indicators, one that is aggregated across the 16 courses and one that 
pertains to each course separately. The first indicator, access rate per user, is the sum of the number of 
times a user accessed any of the four types of activities offered in the course, divided by the number of 
users. We define a user as someone who registers for a course between the dates a course opens and two 
months after the course ended. This indicator recognizes that a user may access a lecture and some 
assessments multiple times. 

The second indicator, the standardized access rate, adjusts for the number of different access 
opportunities a course offers. The rate is calculated as the number of accesses per user divided by the 
number of access opportunities offered. The denominator is effectively the number of “person- 
opportunities” akin to “person-years” or “disability adjusted life-years” used by health economists and 
epidemiologists for over 30 years (Kromann, & Green, 1980; Anand, & Hanson, 1997; Bhupathiraju et 
al., 2014). The standardized access rate sums all of the chances a person has to learn or at least look at 
new content in a course. 

Penn Course Access Rates 

Table 1 (next page) shows that, across the 16 courses, more than 710,000 users had just over 
1,100 opportunities to access lectures, stand-alone quizzes, embedded quizzes and open-ended questions. 
Users took up these opportunities nearly 13 million times. Of the nearly 13 million access behaviors more 
than 9.2 million were tied to lectures and nearly 3.7 million were tied to assessments. 

Across the 16 courses, a given user accessed content an average of 18.2 times. The access rate for lectures 
(13 per user) is higher than the access rate for assessments (5.2 per user). Access rates also vary by type 
of assessment. The rate falls from 3.4 accesses per user for quizzes embedded in lectures, to 1.6 for 
standalone quizzes, to 0.2 for open-ended questions. 
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Table 1 . Access Rates Aggregated Across Courses. 



Number of 

Registrants 

(A) 

Number of 

Access 

Rate (B) / 
(A) 

Number of 

Standardized 

Opportunities 

Times 

Accessed 

(B) 

Access 

Opportunities 

(C) 

Access Rate 
[ (B / A ) / C ] 

Registrants 

Total 

710,385 

12,946,959 

18.2 

1,131 

0.016 

Lectures 


9,251,884 

13.0 

718 

0.018 

Assessments 


3,695,075 

5.2 

413 

0.013 

Total Assessments 


3,695,075 

5.2 

413 

0.013 

Embedded quiz 


2,439,680 

3.4 

186 

0.018 

Standalone quiz 


1,127,228 

1.6 

171 

0.009 

Open-ended 

question 


128,167 

0.2 

56 

0.003 


While interesting, these rates do not take into account the number of instructor-created access 
opportunities in a course. To control for this variation, Table 1 also shows the “standardized access rate.” 
Across all opportunities, the standardized access rate averages 0.016. The standardized access rate is 
somewhat higher for lectures (0.018) than for assessments (0.013). The standardized access rate also 
varies based on type of assessment, falling from 0.018 for quizzes that are embedded in lectures, to 0.009 
for stand-alone quizzes, to 0.003 for open-ended questions. The standardized lecture access rate (0.018) is 
roughly equivalent to the 2% average response rate for direct mail advertising sent to individuals (DMA 
2012 & Yoon & Eckels 2013). 

These aggregate numbers mask variations across courses. Table 2 (next page) shows that the 
average number of times a user accessed course content (that is, the access rate) varies across the courses 
by a factor of 5, ranging from a low of 7.3 in Rationing and Allocating Scarce Medical Resources to 35.4 
in Gamification. 

The average number of times a user accesses course content is not tied to the number of access 
opportunities. Five courses have 100 or more access opportunities (Gamification, Microeconomics, 
Calculus, Mythology, and Modem Poetry) but the access rates for these courses vary from 15.6 to 35.4. 
Among the seven courses with fewer than 50 access opportunities, access rates range from 7.3 to 24.8. 
Gamification and Modem Poetry have the same number (100) of opportunities, but Gamification has a 
much higher access rate (35.4 per user) than Modem Poetry (15.6). When we adjust access rates for the 
number of access opportunities, we continue to see considerable variation across courses. Table 2 shows 
standardized access rates ranging from 0.12 in Calculus and 0.14 in Genome Science to more than 0.6 in 
ADHD, Vaccines, and Pharmacy 101. The five-fold difference between the lowest and the highest 
standardized access rates is important because it shows that some courses engage learners more than 
others. There may be some instructional design choices that produce courses that engage learners more 
than others. These choices (preferred length of video or type of assessment, for example) could be 
revealed by more focused research within the highest and lowest rating courses. Our set of courses is too 
small and too different in design, length, audience, content and degree of difficulty to allow us to identify 
patterns or draw conclusions about the instructional choices that produce the greatest engagement. 
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Table 2. Access Rate for Each Course 


Course 

Number of 
Registrants 
(A) 

Number 
of Times 
Accessed 
(B) 

Access 
(B) / 

Users 
(A) 

Number of 

Opportunities 

(C) 

Standardized 
Access Rate 
[ (B / A ) / C] 

Lecture 

Completion 

Rate 

Gamification 

83,061 

2,942,781 

35.4 

100 

0.35 

18% 

ADHD 

25,995 

645,180 

24.8 

36 

0.69 

15% 

Operations 

99,451 

2,009,708 

20.2 

94 

0.21 


MicroEcon 

35,460 

703,307 

19.8 

123 

0.16 

7% 

Health 

Policy 

30,592 

562,686 

18.4 

42 

0.44 

10% 

Vaccines 

20,689 

365,257 

17.7 

28 

0.63 

14% 

Calculus 

58,170 

981,037 

16.9 

141 

0.12 

6% 

Myth 

63,545 

1,042,949 

16.4 

109 

0.15 

7% 

World 

Music 

35,132 

566,101 

16.1 

73 

0.22 

6% 

ModPo 

39,654 

617,055 

15.6 

100 

0.16 

6% 

PharmlOl 

24,582 

370,331 

15.1 

23 

0.66 


Design 

46,241 

693,331 

15.0 

78 

0.19 

8% 

Cardiac 

44,168 

647,433 

14.7 

41 

0.36 

14% 

Networks 

45,681 

452,757 

9.9 

48 

0.21 

13% 

Genome 

Science 

33,154 

251,593 

7.6 

55 

0.14 


Rationing 

13,228 

96,171 

7.3 

40 

0.18 

7% 


Access or Completion? 

Access rates offer a different view of the potential educational benefits of courses than 
completion rates. In earlier work with this same group of MOOCs, we speculated that relatively low 
completion rates may reflect, in part, user “curiosity, browsing and lack of interest or motivation to 
complete” (Pema et al., 2014 ). We calculated completion rates for populations of “registrants,” those 
who signed up for the course when it opened and at any point up until two months after the last course 
component was released. 1 We defined completion rates as the percentage of registrants accessing a lecture 
in the last week or module of a course and refer to them as “lecture completion rates” to distinguish them 
from completion rates of assessment activities. 


1 Refer to Pema et al (2014), for a complete description of the procedures used to define registrants and calculate 
lecture completion rates. For example, a registrant is a person who signed up for the course when opened and at any 
point up until two months after the last course component was released. 
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Comparing standardized access rates with completion rates for lectures for the 13 courses with 
available data 2 reveals a relationship between the two measures, albeit with some variability (see Figure 
1). The generally positive relationship in Figure 1 is consistent with the assumption that ‘engagement’ (as 
measured here by standardized access rates) is positively related to completion. The implication is that 
standardized access rates can predict completion rates especially when the access rates are low. This 
relationship is illustrated by the clustering of low access and completion rates in the bottom left comer of 
Figure 1. As access rates increase, completion rates also increase but with greater variability as evidenced 
by the spread in the upper right quartile of Figure 1. But there is still a clear relationship between access 
and completion, as the correlation between access and completion rates is 0.81. If one were to construe 
the obtained correlation as being from a sample of some hypothetical population (rather the entire 
population), the correlation would be statistically significant (different from zero) at the alpha=.001 level 
or less. 


Figure 1: Standardized Lecture Access and 
Lecture Completion Rates by Course 
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Discussion 


The data show that a very small share of users takes up available opportunities to access course 
content. The standardized access rates used here compare favorably with those cited for many social 
media sites (e.g., Eyl, 2013) and with response rates to large-scale direct mail marketing programs (DMA, 
2012). These results suggest MOOCs—at least in the form examined in this study—are operating like the 


2 We have lecture completion rates for thirteen of the sixteen courses. For three courses it was not possible to 
determine which lecture was the last in the sequence. See Perna et al (2014) for a more complete discussion of the 
implications of the available data for measurement of completion and other aspects of the analyses. 
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informal environments of such social media sites as Facebook and Instagram despite the presence of more 
formal instructional methods. 

The indicators of access in this study have clear limitations as measures of engagement in 
effective educational practices or educational progress. They do not measure learning, length of time 
spent on an activity, or other potentially useful dimensions of a user’s experience with any set of content 
or assignment. They do not differentiate single and multiple views of the same material. They also do not 
take into account differences in course length or difficulty, or the quality of lecture presentation and 
assessment design. Nonetheless, by assessing the extent to which users are accessing course content rather 
than completing a course, access rates offer a different, more nuanced understanding of the level and 
nature of user engagement with a MOOC. 

Although crude, these indicators of access proposed and used here also offer insights into how 
large numbers of users respond to different types of opportunities to engage with various forms of course 
content. Both the analyses of aggregated data and analyses of data that are disaggregated by course 
suggest users typically are more likely to access lectures than assessments. Users also tend to attempt 
quizzes embedded in lectures more readily than other assessments. In short, an instructor’s choices about 
the array of opportunities appear to influence user activity. These findings build on the conclusion by 
Guo, Kim, and Rubin (2014) that the length of videos in Edx MOOCs influences participation, with 
shorter videos being “much more engaging” (p. 2). The difference between access rates for embedded 
quizzes and stand-alone quizzes suggests that even minimal additional effort or time is a deterrent; stand¬ 
alone quizzes require more effort. This conclusion is consistent with Chi (2009), who notes that the 
demands on the user of activities defined as “constructive” and “interactive” are more taxing, require 
more effort and concentration, or simply require more time—all which reduce ‘take up’ of the learning 
opportunities. For course providers and platfonn managers, indicators like those developed in this study 
may be a useful first step in monitoring the extent to which different types and combinations of activities 
may be providing better opportunities for learning. Access indicators can also be used in marketing 
courses to potential users. Access rates can also show people who are interested in monetizing MOOCs or 
using them as advertising vehicles the actual reach of particular courses and segments of courses. And 
perhaps most importantly for faculty and instructional designers, access rates may inform instructional 
design choices. 

Social media engagement rates are evolving as the industry matures and as technology develops. 
With this evolution new metrics for assessing the effectiveness of large-scale online courses may emerge. 
In the interim, the indicators described in this paper can be used as an empirical foundation for making 
instructional choices that may increase course completion, as well as other measures of user-benefits and 
effectiveness of MOOC offerings. 
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